Gleaning answers from the web∗

A wide variety of valuable textual information resides on the Web, but very little is in a machineunderstandable form such as XML. Instead, the content is usually embedded in HTML markup or other encodings designed for human consumption. The information extraction task is to automatically populate a database with content gleaned from information sources such as Web pages. Wrappers are an important special case of the general information extraction task. A wrapper is a specialized information extraction module tailored for a particular source. For example, a meta-search engine needs a distinct wrapper for each of its underlying search engines. ∗Position paper, AAAI 2002 Spring Symposium on Mining Answers from Texts and Knowledge Bases. Wrappers are usually fairly simple patternmatching programs, because in many applications the documents being processed are highly regular, such as machine-generated HTML text emitted from CGI scripts. Nevertheless, automated approaches to wrappers construction are essential if we want to scale up our applications to integrate data from dozens or thousands of sources. Wrapper induction [8, 12, 10] involves using machine learning techniques to automatically generate wrappers. The input to a wrapper induction algorithm is a set of training examples (sample documents annotated with the information that should be extracted from each), and the output is a wrapper. My contributions include the identification of several classes of wrappers that are both: